A Parallel Implementation of Backpropagation Neural Network on MasPar MP-1

In this paper, we explore the parallel implementation of the backpropagation algorithm with and without hidden layers on MasPar MP-1. This implementation is based on a SIMD architecture, and uses a backpropagation model. Our implementation uses weight batching versus on-line updating of the weights which is used by most serial and parallel implementations of backpropagation. This method results in a smoother convergence to a solution which is comparable to that of the popular method. Versus various systolic array implementations of the backpropagation algorithm which are data driven, and exploit pipelined parallelism, we have developed a true SIMD algorithm which is control driven and exploits two types of parallelism inherent in backpropagation feedforward, layered neural networks, namely architectural parallelism and data parallelism. Most importantly, the processing time is reduced both theoretically and experimentally by the order of 3000 for a network with 7-100-10 input-hidden-output neurons and 1188 training. With this algorithm we have achieved speeds of up to 70 Million Connections per Second (MCS) as throughput of a network stage and over 180 Million Connection UPdates per Second (MCUPS) for training the above network. This is the fastest performance of the standard backpropagation algorithm reported to date.